Portable terminal device and information processing system

ABSTRACT

A portable terminal device in an information processing system and method includes a camera and a microphone. Data of obtained images and voice are transmitted to a server that identifies operations to be executed based on the received voice and image data. The server transmits an identification of one or more results of the plurality of operations to the portable terminal device. When the portable terminal device receives only one result from the server, an operation corresponding to the one result is executed, and when a plurality of results is received, the portable terminal device displays information corresponding to the plurality of results as candidates. Additional voice is captured for selecting one of the plurality of results during the displaying of the information. A determination of one result from the plurality of results is made based on the captured voice, and an operation corresponding to the determined result is executed.

TECHNICAL FIELD

The present invention relates to a portable terminal device and aninformation processing system.

BACKGROUND ART

In recent years, with respect to terminals for a cellular phone, carnavigation system, home AV appliance and the like, user interfaces whichare easily operable by recognizing the voice without using an operationmeans such as a keyboard and mouse have been watched.

Citation List Pat. Literature

Pat. Literature 1: JP-A-2007-041089

SUMMARY OF INVENTION Technical Problem

However, with such terminals that execute operation by voicerecognition, a problem occurs that the terminals cannot be operated inan environment uttering the voice is not preferable such as a libraryand in a very noisy environment.

In view of the problem described above, the object of the presentinvention is to provide a portable terminal device and an informationprocessing system which are more user-friendly even in an environmentwhere uttering the voice is not preferable and a very noisy environment.

SOLUTION TO PROBLEM

As a means for solving the problem described above, for example, theconfigurations described in claims can be employed. As an example, suchconfiguration is used which includes a storage unit that stores lipmovement recognition data, an imaging unit that shoots images includingat least a lip portion of an operator, a lip movement recognition unitthat compares data representing a movement of the lip portion of theoperator obtained from the imaging unit and the lip movement recognitiondata, and thereby recognizes operation to be executed, and a controllerthat executes executable operation according to the lip movementrecognition unit.

ADVANTAGEOUS EFFECTS OF INVENTION

By using the technology of the present invention, a portable terminaldevice and an information processing system which are more user-friendlyeven in an environment where uttering the voice is not preferable and avery noisy environment can be provided.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a configuration example of a portableterminal device according to an embodiment of the present invention.

FIG. 2 is an example of a flowchart of a process for executing operationof a portable terminal by voice recognition or lip movement recognition.

FIG. 3 is a drawing showing an example of an operation option table thatcorresponds to a state of the portable terminal.

FIG. 4 is an example of a flowchart explaining an operationdetermination process.

FIG. 5 is an example of a flowchart explaining a process for capturingthe voice and image of an operator.

FIG. 6(a)is a drawing showing a display example of a portable terminalin a process for capturing the voice and image of the operator.

FIG. 6(b) is a drawing showing another display example of a portableterminal in a process for capturing the voice and image of the operator.

FIG. 7 is an example of a flowchart explaining a voice recognitionoperation determination process.

FIG. 8 is an example of a flowchart explaining a lip movementrecognition operation determination process.

FIG. 9 is a drawing showing an example of the definition of the size oflips.

FIG. 10(a) is a drawing showing an example of lip movement recognitiondata and acquired lip movement data in the lip movement recognitionoperation determination process.

FIG. 10(b) is a drawing showing an example of lip movement recognitiondata and acquired lip movement data in the lip movement recognitionoperation determination process.

FIG. 11(a) is a drawing explaining an example of a lip movementrecognition data renewal process.

FIG. 11(b) is a drawing explaining an example of a lip movementrecognition data renewal process.

FIG. 11(c) is a drawing explaining an example of a lip movementrecognition data renewal process.

FIG. 12 is an example of a flowchart explaining an operation objectdetermination process.

FIG. 13 is an example of a flowchart explaining a lip movementrecognition category determination process.

FIG. 14 is a drawing showing an example of a table of the categoryoptions that correspond to operation.

FIG. 15 is an example of a flowchart explaining a voice recognitioncategory determination process.

FIG. 16 is an example of a flowchart explaining a lip movementrecognition operation object determination process.

FIG. 17 is an example of a flowchart explaining an operation objectselection process by lip movement recognition.

FIG. 18 is a drawing showing a display example of a portable terminal inan operation object selection process.

FIG. 19(a) is a drawing showing an example of lip movement recognitiondata and lip movement acquisition data in the lip movement recognitionoperation object determination process.

FIG. 19(b) is a drawing showing an example of lip movement recognitiondata and lip movement acquisition data in the lip movement recognitionoperation object determination process.

FIG. 19(c) is a drawing showing an example of lip movement recognitiondata and lip movement acquisition data in the lip movement recognitionoperation object determination process.

FIG. 20 is an example of a flowchart explaining a voice recognitionoperation object determination process.

FIG. 21 is the second example of a flowchart explaining a lip movementrecognition operation determination process.

FIG. 22 is an example of a table of a vowel sequence corresponding to anoperation option.

] FIG. 23 is an example of a flowchart explaining a syllable and vowelsequence conversion process.

FIG. 24 is a drawing showing an example of the lip shapes thatcorrespond to the vowels.

FIG. 25 is a drawing showing an example of a table of the size of thelips which corresponds to the vowels.

FIG. 26 is the second example of a flowchart explaining a process forcapturing the voice and image.

FIG. 27(a)is a drawing showing a display example in the process forcapturing the voice and image.

FIG. 27(b)is a drawing showing a display example in the process forcapturing the voice and image.

FIG. 28 is a block diagram showing a configuration example of a portableterminal device according to the second embodiment of the presentinvention.

FIG. 29 is an example of a flowchart explaining an operationdetermination process according to Embodiment 2 of the presentinvention.

FIG. 30 is a block diagram showing a configuration example of a portableterminal device according to the third embodiment of the presentinvention.

FIG. 31 is a block diagram showing a schematic configuration of aninformation processing system including a portable terminal according tothe third embodiment of the present invention and a voice/lip movementrecognition server.

FIG. 32(a) is a flowchart showing an example of the process of theinformation processing system of the present invention.

FIG. 32(b)is a flowchart showing an example of the process of theinformation processing system of the present invention.

FIG. 33 is a flowchart showing an example of the voice/lip movementrecognition process in the voice/lip movement recognition server.

FIG. 34 is a flowchart showing an example of the voice recognitionprocess in the voice/lip movement recognition server.

FIG. 35 is a flowchart showing an example of the lip movementrecognition process in the voice/lip movement recognition server.

DESCRIPTION OF EMBODIMENTS

In the following, examples of embodiments of the present invention willbe described using the drawings.

Embodiment 1

FIG. 1 is a block diagram showing an internal configuration example of aportable terminal device 100 by an embodiment of the present invention.

Here, description will be made with an example of a case of a smartphone. The portable terminal 100 includes a controller 101, a voicerecognition unit 102, a lip movement recognition unit 103, a memory 104,a storage 105, a GPS (Global Positioning System) receiver 106, ageomagnetic sensor 107, an acceleration sensor 108, a gyro sensor 109, abase station communication unit 110, a wireless communication unit 111,a microphone 112, an audio processor 113, a speaker 114, a voice outputunit 115, a touch panel 116, an operation input unit 117, a display 118,an image processing unit 119, an imaging unit 120, and an input/outputI/F 121, and each of them and a bus 150 are connected with each other.

The base station communication unit 110 is a communication interfacesuch as W-CDMA (Wideband Code Division Multiple Access) and GSM(Registered trademark) (Global System for Mobile communications) whichexecute long distance wireless communication with a base station 400.With the base station communication unit 110, it is also possible toconnect with an external network 600 through the base station 500, andto transmit/receive information.

The controller 101 is formed of a CPU (Central Processing Unit) and thelike, and controls respective constituting units and executes variousprocesses by executing programs stored in the memory 104.

The voice recognition unit 102 recognizes the voice of the operatorcaptured from the microphone 112 through the audio processor 113, andrecognizes the operation instructed by the voice. Also, the lip movementrecognition unit 103 recognizes the images including the lips of theoperator captured from the imaging unit 120 through the image processingunit 119, and recognizes the operation instructed by the lip movement ofthe operator.

The controller 101 selects whether the operation is to be executed bythe result recognized from the voice of the operator or the operation isto be executed by the result recognized from the lip movement of theoperator, and executes the operation based on the result selected.

The memory 104 is a flush memory and the like, and stores programs,data, and the like. The data used for recognition by the voicerecognition unit 102 and the lip movement recognition unit 103 describedabove are stored in predetermined areas 104 a, 104 b of the memory 104.

Also, the portable terminal 100 includes the storage 105 such as amemory card, and mail addresses, data of music, video, and photos, andthe like can be stored also in the storage 105.

The programs or the data stored in the memory 104 or the storage 105 canbe renewed and added from time to time by that the base stationcommunication unit 110 executes wireless communication with the basestation and downloads the programs or the data from an external serverand the like not illustrated. Further, it is also possible to renew andadd the data, programs and the like by connecting with an externaldevice 300 such as a personal computer through the input/output I/F 121.

The GPS receiver 106 receives signals from GPS satellites overhead.Thereby, the current position of the portable terminal 100 can bedetected.

The geomagnetic sensor 107 is a sensor that detects the direction towhich the portable terminal 100 faces.

The acceleration sensor 108 is a sensor that detects the acceleration ofthe portable terminal 100, and the gyro sensor 109 is a sensor thatdetects the angular velocity of the portable terminal 100. Theinclination and movement of the portable terminal 100 can be detected indetail by them.

The wireless communication unit 111 is a communication interface thatexecutes wireless communication by a wireless LAN of IEEE802.11a/b/n andthe like, and can connect with the external network 600 through awireless router 500.

The microphone 112 inputs the voice of the outside, and the speaker 111outputs the voice to the outside. The external voice output unit 115outputs the voice by connecting an earphone 200. The voiceinputted/outputted is subjected to audio processing by the audioprocessor 113.

A touch panel 116 includes the operation input unit 117 and the display118. The display 118 is an LCD and the like, displays a picture orimage, and includes the operation input unit 117 such as a touch pad onthe display surface thereof.

The operation input unit 117 is a touch pad of a capacitance type forexample, and detects the touch operation (hereinafter referred to as“touch”) by a finger, touch pen and the like as an operation input.

The imaging unit 120 is a camera and the like. The image displayed onthe display 118 and the image inputted from the imaging unit 120 areprocessed by the image processing unit 119.

The input/output I/F 121 is a USB (Universal Serial Bus) and the likefor example, and is an interface that transmits/receives data to/fromthe external device 300.

Next, an example of a flowchart of a process of the controller 101 isshown in FIG. 2 which is for executing operation by voice recognition orlip movement recognition in the portable terminal device 100.

In FIG. 2 , first, what kind of operation is to be executed isdetermined out of options of executable operations by the state of theportable terminal device 100 (S201). An example of a table of theexecutable operations corresponding to the state of the portableterminal device 100 is shown in FIG. 3 . For example, in a state that ahome screen is displayed, “music reproduction”, “mail” and the likebecome operation options, and, in a state that music is reproduced,“stop”, “forward skip” and the like become operation options. Next, abranch process is executed according to whether selection of the objectof the operation is needed or not (S202). For example, when “musicreproduction” is to be executed as the operation, selection of theobject (music piece and the like) of operation (music reproduction andthe like) becomes necessary such as which music piece is to bereproduced. Also, when “stop” is to be executed as the operation duringmusic reproduction, selection of the object of the operation (stop andthe like) is not necessary. When there is selection of the operationobject (Yes), a process S203 for determining the operation object isexecuted, and the operation (music reproduction for example) is executedfor the selected operation object (music piece for example) (S204). Whenthere is not selection of the operation object in the branch processS202 (No), operation (stop for example) is executed. The table data ofthe operation options corresponding to the state of the portableterminal device shown in FIG. 3 are stored in a memory area 104 c. FIG.4 is a flowchart showing an example of the operation determinationprocess S201.

In FIG. 4 , first, a voice is captured from the microphone 112 throughthe audio processor 113, and the image including at least the lipportion of the operator is captured from the imaging unit 120 throughthe image processing unit 119 (S401). Next, a voice recognitionoperation determination process S402 is executed by the voicerecognition unit 102, and a lip movement recognition operationdetermination process S403 is executed by the lip movement recognitionunit 103. In a branch process S404, whether the voice recognition hasbeen successful in the voice recognition operation determination processS402 is determined by a voice recognition flag. When the voicerecognition has been successful (Yes), which operation is to be executedis determined (S405) based on the result recognized in the voicerecognition operation determination process S402. Next, in a branchprocess 406, the image is captured without that the lip portion departsfrom the imaging range, and whether lip movement data has beensuccessfully acquired is determined by a lip detection flag. When thelip movement data has been successfully acquired (Yes), the lip movementrecognition data of the memory area 104 b is renewed corresponding tothe voice recognition result (S407), manner mode is released (S408), andthe process is finished. In operation of the manner mode release andonward, operation guide by the voice from the speaker 114 (or from theearphone 200 through the external voice output unit 115 when theearphone 200 has been connected), incoming call guiding by sound, andthe like are executed. On the other hand, when it is determined thatacquisition of the lip movement data has failed by the lip detectionflag in the branch process S406 (No), the lip movement recognition dataof the memory area 104 b is not renewed, the manner mode is released(S408), and the process is finished.

When it is determined by the voice recognition flag that the voicerecognition has failed in the branch process S404 (No), whether therecognition has been successful in the lip movement recognitionoperation determination process S403 is determined by a lip movementrecognition flag in a branch process 409. When the lip movementrecognition has been successful (Yes), which operation is to be executedis determined based on the result recognized in the lip movementrecognition operation determination process S403 (S410), the manner modeis set (S411), and the process is finished. In the manner mode, theoutput from the speaker 114 is put off, and operation guide or incomingcall guiding or the like by the screen display without the sound isexecuted. On the other hand, when it is determined that the lip movementrecognition has failed by the lip movement recognition flag in thebranch process S409 (No), the process returns again to the process foracquiring the voice and image (S401).

By the process described above, when the voice recognition operationdetermination process has been successful, the operation is determinedaccording to the voice recognition result, and when the voicerecognition operation determination process has failed and the lipmovement recognition operation determination process has beensuccessful, the operation is determined according to the lip movementrecognition. Also, when the voice recognition has been successful andacquisition of the lip movement data has been successful, the lipmovement recognition data of the memory area 104 b is renewed.

As described above, even when the voice recognition operationdetermination process cannot be executed in a very noisy environment ofthe bustle and the like and an environment of a library and the likewhere uttering is not appropriate, which operation is to be executed canbe determined by executing the lip movement recognition operationdetermination process. Also, setting/releasing of the manner mode can beautomatically executed by the process of the voice recognition and thelip movement recognition.

FIG. 5 is a flowchart showing an example of the process S401 forcapturing the voice and the images including the lip portion.

In FIG. 5 , first, start of capturing of the voice and image isdetermined (S501). As the determination of the start of capturing, forexample, determination is made by whether a predetermined portion M ofthe touch panel 116 of the portable terminal device 100 shown in FIG.6(a) has been touched or not. When it is determined that thepredetermined portion M has been touched (Yes), capturing of the voiceand the image of the lip portion (S502) is started, and the capturedimage is displayed at a predetermined portion W of the display 118 ofthe portable terminal device 100 (S503). Whether the lip portion has notdeparted from the imaging range is detected by the lip movementdetection unit 103 based on the captured image (S504). In a branchprocess S505, a branch process is executed by the lip detection result,and, when the lip portion has not departed from the imaging range asshown in FIG. 6(a) (Yes), for example, the display frame of thepredetermined portion W of the display 118 is colored blue (S506). Whenit is determined that the lip portion has departed from the imagingrange as shown in FIG. 6(b) (No), for example, the display frame of thepredetermined portion W of the display 118 is colored red (S507). In abranch process S508, finish of the capturing of the voice and images isdetermined. As the determination of the finish of the capturing,determination is made by whether the predetermined portion M of thetouch panel 116 of the portable terminal device 100 has been touchedagain or not. When it is determined that the predetermined portion M hasnot been touched (No), the process returns to S502, and the capturing ofthe voice and images is continued. When it is determined that thepredetermined portion M has been touched (Yes), the capturing of thevoice and the images of the lip portion is finished (S509), and theprocess is finished.

By the process described above, the voice and the images of the lipportion are captured. Also, by the display of the captured image and thecolor of the display frame, whether the lip portion has departed fromthe imaging range can be easily determined, and the operator can correctthe imaging position. Further, here, as a method for notifying theoperator of whether the image of the lip portion has departed or notfrom the imaging range, it is configured to change the color of thedisplay frame, however, it is also possible to notify the operator ofthe same by other displaying methods.

Next, an example of a flowchart of the voice recognition operationdetermination process S402 in the voice recognition unit 102 is shown inFIG. 7 .

In FIG. 7 , voice analysis is executed first, and the time seriespattern of the characteristic parameter of an input voice (morespecifically, the time series of a spectrum and cepstrum) is extracted(S701). Next, the likelihood for the voice recognition datacorresponding to the operation options stored in the memory area 104 aas an acoustic model by HMM (Hidden Markov Model) (S702) is calculated.In a branch process S703, when the maximum probability (maximumlikelihood) of the result of the likelihood calculation is equal to orgreater than a predetermined value (here, the value is set to 0.6 as anexample) (Yes), a voice recognition flag is set to OK (S704), anoperation option that gives the maximum probability is determined as therecognition result (S705), and the process is finished. On the otherhand, when it is determined to be No in the branch process S703, it isdetermined that the voice recognition has failed due to a noise and thelike, the voice recognition flag is set to NG (S706), and the process isfinished.

Next, the lip movement recognition operation determination process S403in the lip movement recognition unit 103 will be described using anexample of FIG. 8 .

In the flowchart of FIG. 8 , first, a movement of the lips is detectedfrom the image of the lip movement inputted, and the lip movement datais acquired (S801). As the data of the lip movement, for example, atemporal change of the lateral size X of the lips and the vertical sizeY of the lips is detected as shown in FIG. 9 . When the lip portiondeparts from the imaging range, the lip portion cannot be detected fromthe image inputted, and acquisition of the lip movement data fails inthe lip movement data acquisition process S801, it is determined to beNo in a branch process 802, the lip detection flag and the lip movementrecognition flag are set to NG (S803, S809), and the process isfinished. On the other hand, when the lip movement data has beensuccessfully acquired from the image inputted, it is determined to beYes in the branch process 802, and the lip detection flag is set to OK(S804). Next, the likelihood of the lip movement data acquired and thelip movement recognition data corresponding to the operation optionsstored in the memory range 104 b is calculated (S805). In a branchprocess S806, when the maximum probability (maximum likelihood) of theresult of the likelihood calculation is equal to or greater than apredetermined value (here, the value is set to 0.6 as an example) (Yes),the lip movement recognition flag is set to OK (S807), an operationoption that gives the maximum probability is determined as therecognition result (S808), and the process is finished. On the otherhand, when it is determined to be No in the branch process S806, the lipmovement recognition flag is set to NG (S809), and the process isfinished.

In FIGS. 10(a) and 10(b), examples of the lip movement recognition dataXr(t), Yr(t) and the acquired lip movement data Xd(t), Yd(t) are shown.FIG. 10(a) corresponds to a selection option “o-n-ga-ku-sa-i-se-i (musicreproduction)”, and FIG. 10(b) corresponds to “bide-o-sa-i-se-i (videoreproduction)”. X shows the lateral size of the lips, and Y shows thevertical size of the lips. For example, the size of the lips for “ga”and “sa” corresponding to the vowel “a” is large in both X and Y. On theother hand, the lip size X of “i” and “bi” corresponding to the vowel“i” is comparatively large, whereas Y is small. Thus, from the lipmovement data Xd(t), Yd(t) acquired as the lip movement and the lipmovement recognition data Xr(t), Yr(t) corresponding to the operationoptions stored in the memory 104, an option whose temporal change of thelip size X, Y is closest can be determined as the recognition result.

An example of a lip movement recognition data renewal process S407 willbe described using FIGS. 11(a)-11(c).

FIG. 11(a) shows lip movement recognition data before renewal Yr(t),FIG. 11(b) shows acquired lip movement data Yd(t), and FIG. 11(c) showslip movement recognition data after renewal Yr(t)'. Here, the lipmovement recognition data after renewal Yr(t)' is defined by anexpression below.

Yr(t)′ = Yr(t) + α ⋅ (Yd(t) − Yr(t))

The lip recognition data after renewal Yr(t)' is used as the lipmovement recognition data Yr(t) in the next lip movement recognition.

Here, α is a coefficient that determines the speed at which the lipmovement recognition data converges to the acquired lip movement data.When α=1 for example,

Yr(t)′ = Yd(t)

is fulfilled, and the acquired lip movement data Yd(t) becomes the lipmovement recognition data in the next lip movement recognition. Whenα=0.5,

Yr(t)′ = 0.5 ⋅ (Yd(t) + Yr(t))

is fulfilled, and the average of the acquired lip movement data Yd(t)and the lip movement recognition data before renewal Yr(t) becomes thelip movement recognition data in the next lip movement recognition. Withrespect to the range of α, α that fulfils

0 < α < 1

is selected. As α is larger, the lip movement recognition data convergesto the acquired lip movement data more quickly. FIG. 11(c) shows a caseof α=0.5.

The lip movement recognition data after renewal Xr(t)' is also givensimilarly by the formula below.

Xr(t)′ = Xr(t) + α ⋅ (Xd(t) − Xr(t))

By the process described above, the lip movement recognition data afterrenewal Xr(t)', Yr(t)' are renewed to data closer to the actuallyacquired lip movement than those before renewal, and are used as the lipmovement recognition data Xr(t), Yr(t) in the next lip movementrecognition. By repeating it, the lip movement recognition data Xr(t),Yr(t) which match the lip movement of the operator more closely can beobtained, and the accuracy of the lip movement recognition can beimproved.

By the voice recognition operation determination process or the lipmovement recognition operation determination process described above,which operation is to be executed can be determined.

Next, the process for determining the object of operation (S203) will bedescribed.

In FIG. 12 , an example of a flowchart of the operation objectdetermination process is shown.

In FIG. 12 , first, a voice is captured from the microphone 112 throughthe audio processor 113, and the image including at least the lipportion of the operator is captured from the imaging unit 120 throughthe image processing unit 119 (S1201). Next, a voice recognitioncategory determination process S1202 and a lip movement recognitioncategory determination process S1203 are executed. In a branch processS1204, whether the voice recognition has been successful in the voicerecognition category determination process S1202 is determined by thevoice recognition flag. When the voice recognition has been successful(Yes), the category of the operation object is determined based on theresult recognized in the voice recognition category determinationprocess S1202 (S1205). Next, in a branch process S1206, whether theimage has been captured without that the lip portion departs from theimaging range and acquisition of the lip movement data has beensuccessful is determined by the lip detection flag. When acquisition ofthe lip movement data has been successful (Yes), the lip movementrecognition data of the memory area 104 b are renewed corresponding tothe voice recognition result (S1207), and the process proceeds to thenext process S1210. On the other hand, when it is determined thatacquisition of the lip movement data has failed by the lip movement flagin the branch process S1206 (No), the lip movement recognition data ofthe memory area 104 b is not renewed, and the process proceeds to thenext process S1210.

When it is determined that the voice recognition has failed (No) by thevoice recognition flag in the branch process S1204 (No), whetherrecognition has been successful in the lip movement recognition categorydetermination process S1203 is determined by the lip movementrecognition flag in a branch process 1208. When the lip movementrecognition has been successful (Yes), the category of the operationobject is determined based on the result recognized in the lip movementrecognition category determination process S403 (S1209), and the processproceeds to the next process S1210. On the other hand, when it isdetermined that the lip movement recognition has failed by the lipmovement recognition flag in the branch process S1208 (No), the processreturns again to the process for acquiring the voice and image (S1201).

In S1210, a voice and an image including at least the lip portion of theoperator are acquired again. A voice recognition operation objectdetermination process S1211 and a lip movement recognition operationobject determination process S1212 are executed based on the voice andimage acquired. In a branch process S1213, whether recognition has beensuccessful in the voice recognition operation object determinationprocess S1211 is determined by the voice recognition flag. When thevoice recognition has been successful (Yes), the operation object isdetermined based on the result recognized in the voice recognitionoperation object determination process S1211 (S1214). Next, in a branchprocess S1215, whether the image has been acquired without that the lipportion departs from the imaging range and acquisition of the lipmovement data has been successful is determined by the lip detectionflag. When acquisition of the lip movement data has been successful(Yes), the lip movement recognition data of the memory area 104 b isrenewed corresponding to the voice recognition result (S1216), and theprocess is finished. On the other hand, when it is determined thatacquisition of the lip movement data has failed by the lip detectionflag in the branch process S1215 (No), the lip movement recognition dataof the memory area 104 b is not renewed, and the process is finished.

When it is determined that the voice recognition has failed by the voicerecognition flag in the branch process S1213 (No), whether the lipmovement recognition has been successful in the lip movement recognitionoperation object determination process S1212 is determined by the lipmovement recognition flag in a branch process 1217. When the lipmovement recognition has been successful (Yes), the operation object isdetermined based on the result recognized in the lip movementrecognition operation object determination process S1212 (S1218), andthe process is finished. On the other hand, when it is determined thatthe lip movement recognition has failed by the lip movement recognitionflag in the branch process S1217 (No), the process returns to theprocess for capturing the voice and image again (S1210).

FIG. 13 is a flowchart showing an example of a lip movement recognitioncategory determination process.

In the flowchart of FIG. 13 , first, the lip movement is detected fromthe image of the lip movement inputted, and the lip movement data isacquired (S1301). When the lip portion has departed from the imagingrange, the lip cannot be detected, and acquisition of the lip movementdata has failed in the lip movement data acquisition process S1301, itis determined to be No in the branch process 1302, the lip detectionflag and the lip movement recognition flag are set to NG (S1303, S1309),and the process is finished. On the other hand, when acquisition of thelip movement data from the image of the lips inputted in the lipmovement data acquisition process S1301 has been successful, it isdetermined to be Yes in the branch process 1302, and the lip detectionflag is set to OK (S1304). Next, the likelihood of this lip movementdata acquired and the lip movement recognition data corresponding to theoperation options stored in the memory area 104 b is calculated (S1305).

In FIG. 14 , an example of a table of category options corresponding tooperations is shown. The attribute of the metadata imparted to the datasuch as music and photo is equivalent to the category. For example, torespective music data, data on the attribute (category) such as the nameof the music piece, artist, and album are imparted as the metadata.

In a branch process S1306, when the maximum probability (maximumlikelihood) of the result of the likelihood calculation is equal to orgreater than a predetermined value (here, the value is set to 0.6 as anexample) (Yes), the lip movement recognition flag is set to OK (S1307),an operation category that gives the maximum probability is determinedas the recognition result (S1308), and the process is finished. On theother hand, when it is determined to be No in the branch process S1306,the lip movement recognition flag is set to NG (S1309), and the processis finished.

Next, the voice recognition category determination process S1202 will bedescribed.

FIG. 15 is a flowchart showing an example of the voice recognitioncategory determination process.

In FIG. 15 , first, the voice inputted from the microphone 112 throughthe audio processor 113 is analyzed, and the time series pattern of thecharacteristic parameter of the input voice is extracted (S1501). Next,likelihood for the voice recognition data corresponding to the categoryoptions stored in the memory area 104 a as an acoustic model by HMM(S1502) is calculated. In a branch process S1503, when the maximumprobability (maximum likelihood) of the result of the likelihoodcalculation is equal to or greater than a predetermined value (here, thevalue is set to 0.6 as an example) (Yes), the recognition flag is set toOK (S1504), a category option that gives the maximum probability isdetermined as the recognition result (S1505), and the process isfinished. On the other hand, when it is determined to be No in thebranch process S1503, it is determined that the voice recognition hasfailed due to the noise and the like, the recognition flag is set to NG(S1506), and the process is finished.

By the lip movement recognition category determination process or thevoice recognition category determination process described above, towhich category an operation object belongs can be determined.

Next, a lip movement recognition operation object determination processand a voice recognition operation object determination process fordetermining an operation object that belongs to a determined categorywill be described.

FIG. 16 is a flowchart showing an example of the lip movementrecognition operation object determination process.

In the flowchart of FIG. 16 , first, movement of the lips is detectedfrom the image of the lip movement inputted, and the lip movement datais acquired (S1601). When the lip portion departs from the imagingrange, the lip cannot be detected, and acquisition of the lip movementdata has failed in the lip movement data acquisition process S1601, itis determined to be No in a branch process 1602, the lip detection flagand the lip movement recognition flag are set to NG (S1603, S1611), andthe process is finished. On the other hand, when the lip movement datahas been successfully acquired from the image of the lip portioninputted in the lip movement data acquisition process S1601, it isdetermined to be Yes in the branch process S1602, and the lip detectionflag is set to OK (S1604). To the data such as music and photo stored inthe storage 105, the metadata on the attribute such as the title,artist, and filming date have been imparted. In S1605, the likelihood ofthe lip movement recognition data corresponding to the description ofthe attribute portion of a selected category (lip movement recognitiondata corresponding to the title of a music piece recorded as themetadata of each music data when the name of a music piece for exampleis selected as the category) and the lip movement data acquired iscalculated. In a branch process S1606, when the maximum probability(maximum likelihood) of the result of the likelihood calculation isequal to or greater than a predetermined value (here, the value is setto 0.6 as an example) (Yes), whether there are plural candidate numbers,namely whether there are plural data in which the maximum probabilitybecomes a predetermined value or more, is determined, and a branchprocess is executed (S1607). When the candidate number is one (Yes), thelip movement recognition flag is set to OK (S1608), an operation objectthat gives the maximum probability is determined as the recognitionresult (S1609), and the process is finished. On the other hand, when itis determined to be a case of plural candidate numbers in the branchprocess S1607 (No), an operation object selection process (S1610) isexecuted, and the process is finished. On the other hand, when it isdetermined to be No in the branch process S1606, the lip movementrecognition flag is set to NG (S1611), and the process is finished.

The operation object selection process S1610 will be described using aflowchart of FIG. 17 .

In FIG. 17 , first, plural candidates are displayed on the display 118of the touch pad 116 (S1701). An example of the display is shown in FIG.18 . Here, the example is a case there are three music pieces thatbecome the candidate. Also, the lip movement recognition data and theacquired lip movement data corresponding to them are shown in FIGS.19(a) - 19(c). In this case, a portion where the lip movementrecognition data Xr(t), Yr(t) are almost same to each other is included,and the operation object cannot be determined only by the lip movement.Therefore, characters with different lip shape for selection are addedto the name of the music piece and are displayed (“a”, “i”, and “u”here). Next, the image including at least the lip portion of theoperator is captured from the imaging unit 120 through the imageprocessing unit 119 (S1702). First, the lip movement is detected fromthe image of the lip movement inputted, and the lip movement data isacquired (S1703). When the lip portion departs from the imaging range,the lip cannot be detected and acquisition of the lip movement data hasfailed in the lip movement data acquisition process S1703, it isdetermined to be No in a branch process 1704, the lip detection flag andthe lip movement recognition flag are set to NG (S1705, S1711), and theprocess is finished. On the other hand, when the lip movement data hasbeen successfully acquired from the image inputted in the lip movementdata acquisition process S1703, it is determined to be Yes in the branchprocess S1704, and the lip detection flag is set to OK (S1706). Next,the likelihood of the lip movement data corresponding to this inputtedimage and the lip movement recognition data corresponding to thecharacters added for selection (“a”, “i”, and “u” in the example of FIG.18 ) is calculated (S1707). In a branch process S1708, when the maximumprobability (maximum likelihood) of the result of the likelihoodcalculation is equal to or greater than a predetermined value (here, thevalue is set to 0.6 as an example) (Yes), the lip movement recognitionflag is set to OK (S1709), an option that gives the maximum probabilityis determined as the operation object (S1710), and the process isfinished. On the other hand, when it is determined to be No in thebranch process S1708, the lip movement recognition flag is set to NG(S1711), and the process is finished.

As described above, even when there are plural candidates whose lipmovement is generally same each other, by adding characters or acharacter string of different lip shape, the operation object can bedetermined.

FIG. 20 is a flowchart showing an example of a voice recognitionoperation object determination process S1211.

In FIG. 20 , first, the voice inputted from the microphone 112 throughthe audio processor 113 is analyzed, and the time series pattern of thecharacteristic parameter of the input voice is extracted (S2001). Withrespect to the description of the attribute portion of the data such asmusic and photo (when the name of the music piece is selected forexample as the category, the title of the music piece stored as themetadata of respective music data), the likelihood for the voicerecognition data stored as the acoustic model in the memory area 104 ais calculated (S2002). In a branch process S2003, when the maximumprobability (maximum likelihood) of the result of the likelihoodcalculation is equal to or greater than a predetermined value (here, thevalue is set to 0.6 as an example) (Yes), the voice recognition flag isset to OK (S2004), an operation option that gives the maximumprobability is determined as the recognition result (S2005), and theprocess is finished. On the other hand, when it is determined to be Noin the branch process S2003, the voice recognition flag is set to NG,and the process is finished.

Another embodiment of the lip movement recognition operationdetermination process S405 is shown in FIG. 21 . In the presentembodiment, the lip shape is made to correspond to the vowel, and thelip movement is recognized as a sequence of the vowels.

In FIG. 21 , first, the syllable number N is determined from the imageof the lip movement inputted (S2101). Next, to which vowel the lip shapecorresponding to each syllable corresponds is determined by a syllableand vowel sequence conversion process, and the lip movement is convertedto a vowel sequence corresponding to syllables of N pieces (S2102). Thelikelihood of this vowel sequence corresponding to the inputted imageand the lip movement recognition data expressed by the vowel sequencecorresponding to the operation option stored in the memory 104 b iscalculated (S2103).

In a branch process S2104, when the maximum probability (maximumlikelihood) of the result of the likelihood calculation is equal to orgreater than a predetermined value (here, the value is set to 0.6 as anexample) (Yes), the recognition flag is set to OK (S2105), an operationoption that gives the maximum probability is determined as therecognition result (S2106), and the process is finished. On the otherhand, when it is determined to be No in the branch process S2104, therecognition flag is set to NG (S2107), and the process is finished.

An example of the table of the vowel sequence corresponding to theoperation options stored beforehand in the memory 104 b is shown in FIG.22 . For example, the vowel sequence corresponding to “ongakusaisei(music reproduction)” becomes “o-a-u-a-i-e-i”. The likelihood of thisvowel sequence corresponding to the operation option and the vowelsequence corresponding to the inputted image is calculated, and anoperation option with the largest likelihood is determined as therecognition result. By differentiating the vowel sequence of the row ofthe characters of each operation option, the operation option and thevowel sequence can make one-to-one correspondence, and the operationoption can be determined by the vowel sequence. When recognition is madeby the vowel sequence, the vowel sequence for the operation option isstored, therefore the temporal change of the lip size X and Y is notrequired to be stored as the lip movement recognition data for theoperation option shown in FIGS. 10(a) and 10(b), and the use amount ofthe memory 104 b can be reduced.

In FIG. 23 , an example of a flowchart of a syllable and vowel sequenceconversion process (S2102) is shown.

In FIG. 23 , first, a loop process is started with a parameter I thatdesignates the syllable to be compared with the vowel lip shape beingmade from 1 to the syllable number N (S2301), and a process of S2302 isrepeated to a loop finishing process of S2303. In S2302, the lip shapecorresponding to the Ith syllable of the inputted image and the lipshape corresponding to the vowel in the vowel recognition data stored inthe memory 104 b are compared to each other, and the vowel correspondingto the Ith syllable is determined. By the processes described above, thesyllables of N pieces corresponding to the inputted image are convertedto a vowel sequence.

In FIG. 24 , an example of the lip shapes corresponding to the vowels isshown. Here, the lip shapes that correspond to the vowels of “a”, “i”,“u”, “e”, and “o” of the Japanese language are shown. For example, asshown in the table of FIG. 25 , the size of the vertical width X and thelateral width Y of the lips is expressed in three steps, and are relatedto each vowel. Thereby, the vertical width X and the lateral width Y ofthe lip shape of the inputted image can be obtained, and thecorresponding vowel can be determined according to the table of FIG. 25.

The recognition method by the vowel sequence described above may beapplied not only to the lip movement recognition operation determinationprocess S405 but also to the lip movement recognition categorydetermination process S1104 and the lip movement recognition operationdetermination process S1109.

FIG. 26 is a flowchart showing another embodiment of the process forcapturing a voice and an image including the lip portion.

In FIG. 26 , the point different from the voice/image capturing processof FIG. 5 is that an option display process S510 is added. In FIGS.27(a) - 27(b), an example of the display in the process for capturingthe voice and the image including the lip portion is shown. FIG. 27(a)is the display of an operation option in a state of home in theoperation determination process, and FIG. 27(b)is the display of acategory option in reproducing music in the operation objectdetermination process. Characters with different vowel or a characterstring for selection are added and displayed, and an operation selectionprocess is executed by voice recognition or lip movement recognitionwith respect to the characters or character string portion added.Thereby, because recognition can be executed by short character orcharacter string with different vowel, recognition can be executedeasily and surely.

As described above, by displaying the option on the display 118, theoperation option or the category option is not required to be rememberedeach time, and can be surely selected. However, it may be configuredthat whether the option is to be displayed or not can be set so that theoptions are not displayed when the operator is accustomed to operationof the portable terminal and so on.

Embodiment 2

FIG. 28 is a block diagram showing a configuration example of the secondembodiment of the portable terminal device 100, a same reference signwill be given to a portion having a function same to that of theconfiguration example of FIG. 1 , and description thereof will beomitted.

In comparison with the configuration example of FIG. 1 , the presentembodiment is provided with an operator recognition unit 122 andoperator recognition data with the latter being provided in apredetermined area 104 d of the memory 104, and is coped with a casethere are plural operators who use the portable terminal device 100.

FIG. 29 is a flowchart showing an example of an operation determinationprocess S201 according to Embodiment 2, and a same reference sign isgiven to a process same to that in the flowchart of FIG. 4 .

In FIG. 29 , first, a voice is captured from the microphone 112 throughthe audio processor 113, and an image including at least the lip portionof the operator is captured from the imaging unit 120 through the imageprocessing unit 119 (S401). Next, an operator N who operates isrecognized by the operator recognition unit 122 based on the voiceand/or image captured in the voice/image capturing process S401 and theoperator recognition data stored in the memory area 104 d (S420). As theoperator recognition data, for example, voice recognition data or facerecognition data registered beforehand for logging in to the portableterminal device 100 can be used. After recognizing the operator, a voicerecognition operation determination process S402 is executed by thevoice recognition unit 102, and a lip movement recognition operationdetermination process S403 is executed by the lip movement recognitionunit 103. In a branch process S404, whether the voice recognition hasbeen successful in the voice recognition operation determination processS402 is determined by the voice recognition flag. When the voicerecognition has been successful (Yes), which operation is to be executedis determined (S405) based on the result recognized in the voicerecognition operation determination process S402. Next, in a branchprocess 406, the image is captured without that the lip portion departsfrom the imaging range, and whether lip movement data has beensuccessfully acquired is determined by the lip detection flag. When thelip movement data has been successfully acquired (Yes), the lip movementrecognition data corresponding to the operator N of the memory area 104b is renewed corresponding to the voice recognition result (S421), themanner mode is released (S408), and the process is finished. Inoperations after the manner mode release, operation guide by voice fromthe speaker 114 (or from the earphone 200 through the external voiceoutput unit 115 when the earphone 200 has been connected), incoming callguiding by sound and the like are executed. On the other hand, when itis determined that acquisition of the lip movement data has failed bythe lip detection flag in the branch process S406 (No), the lip movementrecognition data of the memory area 104 b is not renewed, the mannermode is released (S408), and the process is finished.

When it is determined by the voice recognition flag that the voicerecognition has failed in the branch process S404 (No), whether therecognition has been successful in the lip movement recognitionoperation determination process S403 is determined by the lip movementrecognition flag in a branch process 409. When the lip movementrecognition has been successful (Yes), which operation is to be executedis determined based on the result recognized in the lip movementrecognition operation determination process S403 (S410), the manner modeis set (S411), and the process is finished. In the manner mode, theoutput from the speaker 114 is put off, and operation guide or incomingcall guiding or the like by the screen display without the sound isexecuted. On the other hand, when it is determined that the lip movementrecognition has failed by the lip movement recognition flag in thebranch process S409 (No), the process returns again to the process foracquiring the voice and image (S401).

By the processes described above, the lip movement recognition datacorresponding to the operator is renewed for each operator, and the lipmovement recognition data dealing with the individual difference of thelip movement can be obtained. Therefore, because the lip movementrecognition is executed using the lip movement recognition data renewedcorresponding to the operator in the lip movement recognition operationdetermination process S403, even when plural persons use a portableterminal device, the accuracy of lip movement recognition can beimproved.

Further, the lip movement recognition data renewal process correspondingto the operator may be applied not only to the operation determinationprocess S201 but also to the operation object determination process S203in a similar manner.

Embodiment 3

FIG. 30 is a block diagram showing a configuration example of the thirdembodiment of the portable terminal device 100, a same reference signwill be given to a portion having a function same to that of theconfiguration example of FIG. 28 , and description thereof will beomitted. Also, FIG. 31 is a block diagram showing a schematicconfiguration of a voice/lip movement recognition information processingsystem including the portable terminal device 100 of the presentembodiment and a voice/lip movement recognition server 700.

In comparison with the configuration example of FIG. 28 , in theportable terminal device 100 of the present embodiment, the voicerecognition unit 102, the lip movement recognition unit 103, and thevoice recognition data area and the lip movement recognition data areaof the memory 104 are not arranged, and portions corresponding to themare arranged in the voice/lip movement recognition server 700.

In FIG. 31 , the voice/lip movement recognition server 700 includes acontroller 701, a voice recognition unit 702, a lip movement recognitionunit 703, a memory 704, a storage 705, and a communication unit 706, andeach of them are connected to a bus 710 each other.

The communication unit 706 is an interface for connecting to theexternal network 600, and is connected to the portable terminal device100 through the base station 400 or the wireless router 500.

The controller 701 is formed of a CPU and the like, and controlsrespective constituting units and executes various processes byexecuting programs stored in the memory 704.

The voice recognition unit 702 is for recognizing the voice data of theoperator of the portable terminal device 100 obtained through thecommunication unit 706, and converting the same to a character stringcorresponding to the voice data. Also, the lip movement recognition unit703 is for recognizing the lip movement from the image data of theoperator of the portable terminal device 100 obtained through thecommunication means 706, and converting the same to a character stringcorresponding to the image data. The controller 701 transmits the resultrecognized from the voice of the operator or the result recognized fromthe lip movement of the operator to the portable terminal device 100through the communication unit 706.

The memory 704 is a flush memory and the like, and stores programs,data, and the like. The storage 705 is an SSD (Solid State Device) or ahard disk, and the data used for recognition in the voice recognitionunit 702 and the lip movement recognition unit 703 described above isstored in predetermined areas 705 a and 705 b of the storage 705.

FIG. 32 is a flowchart showing an example of the process of thecontroller 101 and the controller 701 in the information processingsystem that includes the portable terminal device 100 and the voice/lipmovement recognition server 700 of FIG. 31 .

In FIG. 32 , first, in the portable terminal device 100, the voice iscaptured from the microphone 112 through the audio processor 113, andthe image including at least the lip portion of the operator is capturedfrom the imaging unit 120 through the image processing unit 119 (S3201).The operator N who operates is recognized by the operator recognitionunit 122 based on the voice and/or image captured in the voice/imagecapturing process S3201 and the operator recognition data stored in thememory area 104 d (S3202).

Next, the data of the voice and image captured is transmitted to thevoice/lip movement recognition server 700 through the base stationcommunication unit 110 or the wireless communication unit 111 (S3203).In the voice/lip movement recognition server 700, a voice and lipmovement recognition process S3204 is executed based on the data of thevoice and image received, and the recognition result is transmitted tothe portable terminal device 100 through the communication unit 706(S3205). In S3206, a branch process is executed based on the voicerecognition flag and the lip movement recognition flag of therecognition result transmitted from the voice/lip movement recognitionserver 700, and, when the voice recognition and the lip movementrecognition have failed (No), the process returns to the voice/imagecapturing process S3201. When the voice recognition or the lip movementrecognition has been successful (Yes), an operation determinationprocess S3207 is executed, which operation is to be executed isdetermined based on the likelihood of the option data on the operationsshown in FIG. 3 stored in the memory 104 c and the recognition result,and, when there exists an operation option corresponding to therecognition result, a determination success/failure flag is set to OK.In a branch process 3408, a branch process is executed by thedetermination success/failure flag, and, when an operation optioncorresponding to the recognition result does not exist (No), the processreturns to the voice/image capturing process S3201. When there exists anoperation option corresponding to the recognition result (Yes), a branchprocess is executed based on the voice recognition flag in a branchprocess S3209. The manner mode is released (S3210) when the voicerecognition has been successful (Yes), and the manner mode is set(S3211) when the voice recognition has failed (No).

Next, a branch process is executed according to whether selection of theobject of the operation is needed or not (S3212). For example, when“music reproduction” is to be executed as the operation, selection ofthe object (music piece and the like) of operation (music reproductionand the like) becomes necessary such as which music piece is to bereproduced. Also, when “stop” is to be executed as the operation duringmusic reproduction, selection of the object of the operation (stop andthe like) is not necessary. When there is not selection of the operationobject (No), the determined operation is executed (S3228). When it isdetermined in the branch process S3212 that there is an operationselection (Yes), a voice/image capturing process S3213 is executed, andthe data of the voice and image captured is transmitted to the voice/lipmovement recognition server 700 (S3214). In the voice/lip movementrecognition server 700, a voice and lip movement recognition processS3215 is executed based on the data of the voice and image received, andthe recognition result is transmitted to the portable terminal device100 (S3216). In S3217, a branch process is executed based on the voicerecognition flag and the lip movement recognition flag of therecognition result transmitted from the voice/lip movement recognitionserver 700, and, when the voice recognition and the lip movementrecognition have failed (No), the process returns to the voice/imagecapturing process S3213. When the voice recognition or the lip movementrecognition has been successful (Yes), an operation categorydetermination process S3218 is executed, the category of the operationobject is determined based on the likelihood of category option datacorresponding to the operation as shown in FIG. 13 stored in the memory104 c and the recognition result, and, when there exists a categoryoption corresponding to the recognition result, the determinationsuccess/failure flag is set to OK. In a branch process 3418, a branchprocess is executed by the determination success/failure flag, and, whenthe category option corresponding to the recognition result does notexist (No), the process returns to the voice/image capturing processS3213. When there exists the category option corresponding to therecognition result (Yes), a voice/image capturing process S3220 isexecuted, and the data of the voice and image captured is transmitted tothe voice/lip movement recognition server 700 (S3221). In the voice/lipmovement recognition server 700, a voice and lip movement recognitionprocess S3222 is executed based on the data of the voice and imagereceived, and the recognition result is transmitted to the portableterminal device 100 (S3223). In S3224, a branch process is executedbased on the voice recognition flag and the lip movement recognitionflag of the recognition result transmitted from the voice/lip movementrecognition server 700, and, when the voice recognition and the lipmovement recognition have failed (No), the process returns to thevoice/image capturing process 3220. When the recognition has beensuccessful (Yes), an operation object determination process S3225 isexecuted. In a memory 104 e of the portable terminal 100, history dataof each operator has been stored, and the history of the words and thelike of the retrieval object when internet retrieval was performed byvoice recognition has been stored. Also, to the data such as music andphoto stored in the storage 105 of the portable terminal device 100,metadata on the attribute such as the title, artist, and filming datehas been imparted. The operation object is determined based on thelikelihood of the description of the history data stored in the memory104 e and the attribute portion of the category determined by theoperation category determination process S3218 (for example, thedescription corresponding to the title of the music piece recorded asthe metadata of each music data when the name of the music piece hasbeen selected as the category) and the recognition result, and, whenthere exists an operation object corresponding to the recognitionresult, the determination success/failure flag is set to OK. In a branchprocess 3426, a branch process is executed by the determinationsuccess/failure flag, and, when an operation object corresponding to therecognition result does not exist (No), the process returns to thevoice/ image capturing process S3220. When there exists an operationobject corresponding to the recognition result (Yes), the determinedoperation object is added/renewed with respect to the history datacorresponding to the operator N stored in the memory 104 e (S3227), andoperation is executed for the determined operation object (S3228).

An example of the flowchart of the voice/lip movement recognitionprocesses S3204, S3215, and S3222 is shown in FIG. 33 . In FIG. 33 ,first, a voice recognition process S3301 is executed by the voicerecognition unit 702 based on the voice data of the operator and theimage data including at least the lip portion acquired through thecommunication unit 706, and a lip movement recognition process S3302 isexecuted by the lip movement recognition unit 703. In a branch processS3303, whether the voice recognition has been successful is determinedby the voice recognition flag in the voice recognition process S3301.When the voice recognition has failed (No), the process is finished.When the voice recognition has been successful (Yes), whether the imagehas been captured without that the lip portion departs from the imagingrange and acquisition of the lip movement data has been successful isdetermined by the lip detection flag in a branch process S3304. Whenacquisition of the lip movement data has been successful (Yes), a branchprocess is executed by whether or not there is the lip recognition datacorresponding to the character string obtained by the voice recognition.When there is the lip recognition data corresponding to the characterstring obtained by the voice recognition (Yes), the lip recognition dataof the storage area 705 b corresponding to the character string isrenewed (S3306), and, when the lip recognition data does not exist (No),the lip recognition data corresponding to the character string obtainedby the voice recognition is added to the storage area 705 b (S3307), andthe process is finished. On the other hand, when it is determined in thebranch process S3304 that acquisition of the lip movement data hasfailed (No), the lip movement recognition data is not renewed, and theprocess is finished.

By the processes described above, when the voice recognition has beensuccessful and acquisition of the lip movement data has been successful,renewal and addition of the lip movement recognition data correspondingto the voice recognition result are executed.

An example of the flowchart of the voice recognition process S3301 isshown in FIG. 34 . In FIG. 34 , first, the voice analysis is executed,and the time series pattern of the characteristic parameter of the inputvoice is extracted (S3401). Next, the likelihood for the voicerecognition data stored in the predetermined area 705 a of the storageas an acoustic model by HMM is calculated (S3402). In a branch processS3403, when the maximum probability (maximum likelihood) of the resultof the likelihood calculation is equal to or greater than apredetermined value (here, the value is set to 0.6 as an example) (Yes),the voice recognition flag is set to OK (S3404), the voice recognitiondata that gives the maximum probability is made the recognition result,and the process is finished. On the other hand, when it is determined tobe No in the branch process S3403, it is determined that the voicerecognition has failed due to the noise and the like, the voicerecognition flag is set to NG (S3405), and the process is finished.

Next, the lip movement recognition process S3302 will be described usingan example of FIG. 35 .

In the flowchart of FIG. 35 , first, the lip movement is detected fromthe image inputted, and the lip movement data is acquired (S3501). Whenthe lip portion has departed from the imaging range, the lip portioncannot be detected from the image inputted, and acquisition of the lipmovement data has failed in the lip movement data acquisition processS3501, it is determined to be No in a branch process 3702, the lipdetection flag and the lip movement recognition flag are set to NG(S3503, S3508), and the process is finished. On the other hand, whenacquisition of the lip movement data from the image inputted has beensuccessful, it is determined to be Yes in a branch process S3502, andthe lip detection flag is set to OK (S3504). Next, the likelihood of thelip movement data acquired and the lip movement recognition data storedin the predetermined area 705 b of the storage is calculated (S3505). Ina branch process S3506, when the maximum probability (maximumlikelihood) of the result of the likelihood calculation is equal to orgreater than a predetermined value (here, the value is set to 0.6 as anexample) (Yes), the lip movement recognition flag is set to OK (S3507),and the lip movement recognition data that gives the maximum probabilityis made the recognition result, and the process is finished. On theother hand, when it is determined to be No in the branch process S3506,the lip movement recognition flag is set to NG (S3508), and the processis finished.

Although it is configured in the embodiment described above that thevoice and the image data captured in the portable terminal device 100are transmitted to the voice/lip movement recognition server 700, it mayalso be configured that the voice analysis for extracting the timeseries pattern of the characteristic parameter of the input voice isexecuted in the portable terminal device 100 and the result of detectingthe lip movement from the image inputted and acquiring the lip movementdata is transmitted to the voice/lip movement recognition server 700.Thereby, the data amount transmitted from the portable terminal device100 to the voice/lip movement recognition server 700 can be reduced, andthe processing time can be reduced.

In the embodiments described above, by executing the voice recognitionand the lip movement recognition by the voice/lip movement recognitionserver 700, the lip movement recognition data is renewed based on thedata of the voice and lip movement of a number of operators, andtherefore the accuracy of the lip movement recognition can be furtherimproved. Also, by arranging the history data for each operator, addingthe words and the like newly used in the voice recognition as thehistory data, and utilizing the history data in the lip movementrecognition, lip movement recognition of the words with high usefrequency for each operator of the portable terminal device becomespossible.

Also, the embodiments described above were described in detail in orderto facilitate easy understanding of the present invention, and thepresent invention is not necessarily limited to those including allconfigurations described. For example, although the voice recognitiondata, lip movement recognition data, and option data were stored in thememory 104 in the first and second embodiments, they may be stored inthe storage 105. Further, a part of the configuration of an embodimentcan be replaced by a configuration of another embodiment, and aconfiguration of an embodiment can be added with a configuration ofanother embodiment. Furthermore, with respect to a part of theconfiguration of each embodiment, it is possible to effect addition,deletion, and replacement of other configurations.

Also, with respect to each configuration, function, processor,processing means and the like described above, a part or all thereof maybe achieved by hardware by designing by an integrated circuit and so onfor example. Further, each configuration, function, and the likedescribed above may be achieved by software by that a processorinterprets and executes a program that achieves the function of them.Information such as a program, table, and file achieving each functioncan be placed in the memory 104 and the storage 105.

Also, with respect to the control line and information line, thoseconsidered to be necessary for explanation have been shown, and allcontrol lines and information lines applicable to products have notnecessarily been shown. Practically, it may be thought that almost allconfigurations are connected to each other.

REFERENCE SIGNS LIST

100: portable terminal device, 101: controller, 102: voice recognitionunit, 103: lip movement recognition unit, 104: memory, 105: storage,110: base station communication unit, 111: wireless communication unit,112: microphone, 113: audio processor, 114: speaker, 115: external voiceoutput unit, 116: touch panel, 117: operation input unit, 118: display,119: image processing unit, 120: imaging unit, 122: operator recognitionunit, 400: base station, 500: wireless router, 600: external network,700: voice/lip movement recognition server, 701: controller, 702: voicerecognition unit, 703: lip movement recognition unit, 705: storage, 706:communication unit

1. A portable terminal device comprising: a camera that captures imagesof an operator; a microphone that captures voice instructions of theoperator; a controller which is programmed to execute a plurality ofoperations; a communication interface that transmits and receives datawith an external server; and wherein the controller is furtherprogrammed to: when the images are obtained from the camera and a singlevoice instruction is obtained from the microphone, perform a facerecognition based on the images and thereafter control the communicationinterface to transmit data of the obtained single voice instruction tothe external server, and when the communication interface receivesinformation from the external server including one or more resultsidentified by the external server based on the transmitted data,wherein, when the communication interface receives only one result foroperation identified in response to the transmitted data by the externalserver as operation based on the single voice instruction, thecontroller is further programmed to execute an operation correspondingto the one result, wherein, when the communication interface receives aplurality of results for operation identified in response to thetransmitted data by the external server as operation based on the singlevoice instruction, the controller is further programmed to: display aplurality of operation options to be operable by the portable terminaldevice, the plurality of operation options relating to each of theplurality of results for operation based on the single voiceinstruction, capture additional voice instruction for selecting oneoperation option from the plurality of the operation optionscorresponding to the single voice instruction to be operable by theportable terminal device during displaying of the plurality of operationoptions based on the single voice instruction, determine one operationoption from the plurality of the operation options to be operable by theportable terminal device based on the captured voice instruction, andexecute an operation corresponding to the determined one operationoption.
 2. The portable terminal device according to claim 1, whereinthe displayed plurality of operation options corresponding to theplurality of results are displayed as characters or a character string.3. The portable terminal device according to claim 2, wherein thecontroller is further programmed to determine the one operation optionbased on the captured voice corresponding to the characters or thecharacter string.
 4. The portable terminal device according to claim 1,wherein when the images are obtained from the camera and the voice isobtained from the microphone, the controller is further programmed toidentify if the operator is a specific operator based on the obtainedimages, and when the operator is identified as the specific operator,the obtained voice are transmitted to the external server.
 5. Theportable terminal device according to claim 1, further comprising: aspeaker, wherein the controller is further programmed to: restrictoutput from the speaker, and control the speaker to output audio of avoice associated with the operation based on the information receivedfrom the external server when the output from the speaker is restricted.6. An information processing system comprising: a portable terminaldevice; and a server connected over a network to the portable terminaldevice, wherein the portable terminal device includes: a camera thatcaptures images of an operator; a microphone that captures voiceinstructions of the operator; a first controller which is programmed toexecute a plurality of operations; a first communication interface thattransmits and receives data with the server; and wherein the firstcontroller is further programmed to: when the images are obtained fromthe camera and a single voice instruction is obtained from themicrophone, perform a face recognition based on the images andthereafter control the first communication interface to transmit data ofthe obtained single voice instruction to the server, wherein the serverincludes: a second communication interface that receives the data of theobtained single voice instruction transmitted to the server; and asecond controller which is programmed to execute a plurality ofoperations, wherein the second controller is further programmed to:identify one or more of the operations to be executed based on thereceived single voice instruction data, and control the secondcommunication interface to transmit an identification of one or moreresults of the plurality of operations to the portable terminal device,wherein the first communication interface receives information from theserver including the one or more results identified by the server,wherein, when the first communication interface receives only one resultfor operation identified in response to the transmitted data by theserver as operation based on the single voice instruction, the firstcontroller is further programmed to execute an operation correspondingto the one result, wherein, when the first communication interfacereceives a plurality of results for operation identified in response tothe transmitted data by the server as operation based on the singlevoice instruction, the first controller is further programmed to:display a plurality of operation options to be operable by the portableterminal device, the plurality of operation options relating to each ofthe plurality of results for operation based on the single voiceinstruction, capture additional voice instruction for selecting oneoperation option from the plurality of the operation optionscorresponding to the single voice instruction to be operable by theportable terminal device during displaying of the plurality of operationoptions based on the single voice instruction, determine one operationoption from the plurality of the operation options to be operable by theportable terminal device based on the captured voice instruction, andexecute an operation corresponding to the determined one operationoption.
 7. The information processing system according to claim 6,wherein the displayed plurality of operation options corresponding tothe plurality of results are displayed as characters or a characterstring.
 8. The information processing system according to claim 7,wherein the controller is further programmed to determine the oneoperation option based on the captured voice corresponding to thecharacters or the character string.
 9. The information processing systemaccording to claim 6, wherein when the images are obtained from thecamera and the voice is obtained from the microphone, the firstcontroller is further programmed to identify if the operator is aspecific operator based the obtained images, and when the operator isidentified as the specific operator, the obtained voice are transmittedto the server.
 10. The information processing system according to claim6, wherein the portable terminal further comprises: a speaker, whereinthe first controller is further programmed to: restrict output from thespeaker, and control the speaker to output audio of a voice associatedwith the operation based on the information received from the serverwhen the output from the speaker is restricted.
 11. An informationprocessing method comprising: capturing images of an operator with acamera of a portable terminal device; capturing voice instructions ofthe operator with a microphone of the portable terminal device;transmitting and receiving data between the portable terminal device anda server connected over a network to the portable terminal device;wherein a first controller of the portable terminal device is programmedto execute a plurality of operations, including: when the images areobtained from the camera and a single voice instruction is obtained fromthe microphone, performing a face recognition based on the images andthereafter transmitting the single voice instruction obtained from themicrophone to the server; wherein a second controller of the server isprogrammed to execute a plurality of operations, including: receivingthe data of the obtained voice transmitted to the server from theportable terminal device, wherein the second controller of the server isfurther programmed to: identify one or more of the operations to beexecuted based on the received single voice instruction data, andtransmit an identification of one or more results of the plurality ofoperations to the portable terminal device, wherein the portableterminal device receives information from the server including the oneor more results identified by the server, wherein, when the portableterminal device receives only one result for operation identified inresponse to the transmitted data by the server as operation based on thesingle voice instruction, the first controller of the portable terminaldevice is programmed to execute an operation corresponding to the oneresult, wherein, when the portable terminal device receives a pluralityof results for operation identified in response to the transmitted databy the server as operation based on the single voice instruction, thefirst controller of the portable terminal device is further programmedto: display a plurality of operation options to be operable by theportable terminal device corresponding to the plurality of results foroperation, capture additional voice instruction for selecting oneoperation option from the plurality of the operation optionscorresponding to the single voice instruction to be operable by theportable terminal device during displaying the plurality of operationoptions based on the single voice instruction, determine one operationoption from the plurality of the operation options to be operable by theportable terminal device based on the captured voice instruction, andexecute an operation corresponding to the determined one operationoption.
 12. The information processing method according to claim 11,wherein the displayed plurality of operation options corresponding tothe plurality of results are displayed as characters or a characterstring.
 13. The information processing method according to claim 12,wherein the controller is further programmed to determine the oneoperation option based on the captured voice corresponding to thecharacters or the character string.
 14. The information processingmethod according to claim 11, wherein when the images are obtained fromthe camera and the voice is obtained from the microphone, the firstcontroller is further programmed to identify if the operator is aspecific operator based the obtained images, and when the operator isidentified as the specific operator, the obtained voice are transmittedto the server.
 15. The information processing method according to claim11, wherein the first controller of the portable terminal further isfurther programmed to: restrict output from a speaker of the portableterminal device, and control the speaker to output audio of a voiceassociated with the operation based on the information received from theserver when the output from the speaker is restricted.